Systems and methods for indicating user performance in launching a basketball toward a basketball hoop

ABSTRACT

Methods and apparatus for determining a trajectory of a axisymmetric object in 3-D physical space using a digital camera which records 2-D image data are described. In particular, based upon i) a characteristic length of the axisymmetric object, ii) a physical position of the camera determined from sensors associated with the camera (e.g., accelerometers) and iii) captured 2-D digital images from the camera including a time at which each image is generated relative to one another, a position, a velocity vector and an acceleration vector can be determined in three dimensional physical space for axisymmetric object objects as a function of time. In one embodiment, the method and apparatus can be applied to determine the trajectories of objects in games which utilize axisymmetric object objects, such as basketball, baseball, bowling, golf, soccer, rugby or football.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority to and is a continuation of U.S. patentapplication Ser. No. 16/777,838, entitled, “Systems and Methods forIndicating User Performance in Launching a Basketball Toward aBasketball Hoop,” and filed on Jan. 30, 2020, which is incorporatedherein by reference in its entirety and for all purposes. U.S. patentapplication Ser. No. 16/777,838 claims priority to and is a continuationof U.S. patent application Ser. No. 15/624,527, entitled “True SpaceTracking of Axisymmetric Object Flight Using Diameter Measurement” andfiled on Jun. 15, 2017, which is incorporated herein by reference in itsentirety and for all purposes. U.S. patent application Ser. No.15/624,527 claims priority to and is a continuation of U.S. Pat. No.9,697,617, entitled “True Space Tracking of Axisymmetric Object FlightUsing Image Sensor” and filed on Dec. 22, 2014, which is incorporated byreference in its entirety and for all purposes. U.S. Pat. No. 9,697,617claims priority to and is a continuation of U.S. Pat. No. 8,948,457,entitled “True Space Tracking of Sphere Flight Using DiameterMeasurement” and filed on Jun. 18, 2013, which is incorporated byreference in its entirety and for all purposes. U.S. Pat. No. 8,948,457claims priority under 35 U.S.C. § 119(e) to U.S. Provisional PatentApplication No. 61/808,061, entitled “True Space Tracking of SphereFlight Using Diameter Measurement” and filed on Apr. 3, 2013, which isincorporated by reference in its entirety and for all purposes.

FIELD OF THE INVENTION

The present invention relates generally to devices and systems forsports training and entertainment and more specifically tocharacterizing ball motion in sporting environments using 2-D imagedata.

BACKGROUND

There are many different games which involve use of an essentiallyspherical object or ball. A few examples include basketball, soccer,golf, baseball, softball, tennis, volleyball, racket ball, water-polo,lacrosse, bowling, shot-put, jai alai, polo, handball ping-pong, fieldhockey, dodgeball, billiards, cricket, kickball, wiffleball, skeeball,pinball and foosball. In each game, a ball is moved in some manner,under control of a game participant, from location to location,sometimes colliding with, passing through or over other objectsdepending on the rules of the game.

For ball-based games, ball control is integral to the play of the games.Thus, information which quantifiable characterizes the motion of theball is of interest to participants and spectators alike. As an example,participants can be interested in the information which characterizesthe motion of the ball for the purposes of assessing and improving theirperformance in the game. Whereas for spectators, the information mayenhance the entertainment derived from viewing the sport.

Devices which are often used to characterize ball motion can beexpensive, can require specialized set-up and can require complexcalibration. Thus, the average person doesn't have access to informationwhich characterizes ball motion in most situations. In view of theabove, improved methods and apparatus for characterizing ball motion aredesired.

SUMMARY

Methods and apparatus for determining a trajectory of a spherical objectin 3-D physical space using a digital camera which records 2-D imagedata are described. Based upon a known size of the ball in 3-D physicalspace, a position in the 3-D physical space of the camera which isdetermined from sensors associated with the camera (e.g.,accelerometers) and the 2-D image data which provides pixel data, atrajectory of the spherical object in 3-D physical space as a functionof time can determined. The method and apparatus can be applied todetermine the trajectories of balls used in sports, such as basketball,baseball, bowling, golf and tennis. In particular embodiments, themethod and apparatus can be implemented on a mobile computing deviceincluding the camera. Information derived from the trajectories can beused to improve performance of participants in the sports, enhance theentertainment value associated with viewing the sports or enhance theentertainment value of video games derived from the sports.

One aspect of the methods and apparatus is related to a non-transitorycomputer readable medium for storing a computer program used by acomputing device where the computer program is executed by the computingdevice to generate a three dimensional (3-D) trajectory of a ball fromtwo dimensional (2-D) digital image data. The ball can be a sphericalobject or a non-spherical object, such as a football having anelliptical cross section. The computer readable medium can include 1)computer code for receiving a sequence of 2-D digital images from adigital camera associated with the computing device; 2) computer codefor receiving orientation data which is used to determine or to specifyan orientation of a lens of the digital camera in 3-D physical space,said lens configured to receive light used to generate the 2-D digitalimages; 3) computer code for transforming pixel data in each of the 2-Ddigital images based upon the determined orientation of the lens; 4)computer code for identifying a 2-D representation of a ball captured inthe transformed pixel data associated with each of the 2-D digitalimages; 5) computer code for determining pixel coordinates of the 2-Drepresentation of the ball in each of the 2-D digital images; 6)computer code for determining a characteristic length of the 2-Drepresentation of the ball measured in pixels in each of the 2-D digitalimages wherein the characteristic length of the 2-D representationvaries within the sequence of the 2-D digital images; 7) computer codefor determining a distance to the identified 2-D representation of theball from the lens based upon the characteristic length in the pixels,said distance measured in units associated with 3-D physical space; and8) computer code for determining 3-D coordinates of the ball as afunction of time in the 3-D physical space based upon the determineddistance and the pixel coordinates of the 2-D representation of the ballcaptured in the transformed pixel data associated with each of the 2-Ddigital images.

BRIEF DESCRIPTION OF THE DRAWINGS

The included drawings are for illustrative purposes and serve only toprovide examples of possible structures and process steps for thedisclosed inventive systems and methods for providing game services toremote clients. These drawings in no way limit any changes in form anddetail that may be made to the invention by one skilled in the artwithout departing from the spirit and scope of the invention.

FIG. 1 is a representation of a 2-D composite image including atrajectory of a basketball in flight in accordance with the describedembodiments.

FIG. 2 is a representation of a 2-D composite image including atrajectory of baseball in flight in accordance with the describedembodiments.

FIG. 3 is a system diagram including a 2-D digital camera in accordancewith the described embodiments.

FIG. 4A is an illustration of a digital camera setup in accordance withthe described embodiments.

FIG. 4B is an illustration showing a 3-D object projected onto a 2-Dimage captured by a camera in accordance with the described embodiments.

FIG. 5 is a flow chart 502 including one or more calibration steps whichcan be used to determine a trajectory of an axisymmetric in 3-D physicalspace using 2-D image data.

FIG. 6 is a flow chart of a method for determining a 3-D trajectory of aspherical object using 2-D image data in accordance with the describedembodiments.

FIG. 7 is an illustration of a parabolic trajectory and planartrajectory in accordance with the described embodiments.

FIG. 8A is an illustration of a prolate spheroid shaped object invarious orientations in accordance with the described embodiments.

FIG. 8B is an illustration of a Frisbee shaped object in variousorientations in accordance with the described embodiments.

FIG. 8C is a representation of a 2-D composite image including atrajectory of a football in flight in accordance with the describedembodiments.

FIG. 9 is a block diagram of a system including true space tracking ofaxisymmetric object flight using 2-D image data in accordance with thedescribed embodiments.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

The present invention will now be described in detail with reference toa few preferred embodiments thereof as illustrated in the accompanyingdrawings. In the following description, numerous specific details areset forth in order to provide a thorough understanding of the presentinvention. It will be apparent, however, to one skilled in the art, thatthe present invention may be practiced without some or all of thesespecific details. In other instances, well known process steps and/orstructures have not been described in detail in order to notunnecessarily obscure the present invention.

Mobile computing platforms are becoming ubiquitous as reflected in theincreasing number of smart phones and tablet computers being sold. Mostof these mobile computing platforms include significant computationalcapabilities and at least one high resolution 2-D digital camera capableof capturing still and video images. In addition, many of these mobilecomputing platforms have integrated position sensing accelerometers. Aswill described in more details as follows, the computing, 2-D imagecapture and sensing capabilities of mobile computing devices can beapplied to three dimensional ball tracking.

In alternate embodiments, three dimensional ball tracking can beimplemented using an image capture device and computing deviceconfigured to operate at a fixed position, such as mounted to wall. Theorientation and/or position of the image capture device can bedetermined using sensor devices coupled to the image capture device. Inaddition, the orientation and/or position of the image captured devicecan be directly determined. For example, when mounted to a wall, theorientation and position of the image capture device can be measured bya user and input into the system. These measurements may be provided aspart an initial device calibration.

In sports, three dimensional ball tracking is useful for many purposesincluding training, broadcast enhancement, social sharing and videogames. For example, with readily handily mobile computing devices, 3-Dball tracking can be used to provide feedback information which allows aparticipant to improve in a chosen sporting activity. Broadcastenhancement may involve providing trajectory information associated with3-D ball tracking, which allows a viewer of a sporting activity, such asa coach or parent, to assess the performance of individualsparticipating in the activity.

Social sharing can involve a person using their mobile computing deviceto measure 3-D ball movement associated with their sporting activities,comment on plays and post them to a social media site. Video game playcan involve linking a person's gaming activities on a living room gamingconsole to outside real-life participation in a sport. For example, 3-Dball tracking can be used to assess a person's real-life skill at asport and then information from the assessment can be used to adjustvideo game parameters which affect their performance within the videogame.

In more detail, with respect to FIGS. 1 and 2 , method and apparatusused to characterize ball motion in 3-D space using 2-D image data and aknown size of the ball are briefly described. In particular, applicationof a method and an apparatus for determining a 3-D trajectory of abasketball or baseball are introduced with respect to FIGS. 1 and 2 ,respectively. With respect to FIG. 3 , 2-D image data capture andmanipulation are briefly described. With respect to FIGS. 4A, 4B and 5reference frames and coordinate transformations derived from positionsensing used to work with the 2-D image data are described. A firstmethod of determining a 3-D trajectory of a ball in physical space using2-D image data and accelerometer data is discussed with respect to FIG.6 . A second method of determining a 3-D trajectory of a ball inphysical space using 2-D image data is discussed with respect to FIG. 7. With respect to FIGS. 8A, 8B and 8C, trajectory determination for somenon-spherical objects is discussed. Finally, a system which generatesand utilizes the 3-D trajectory data in various applications isdiscussed with respect to FIG. 9 .

FIG. 1 is a representation of a 2-D composite image 100 including atrajectory 128 of a basketball in flight. A 2-D composite image 100 witha trajectory of a ball can be generated by capturing a series of 2-Dimages from a video camera at a determined position, such as a fixedposition, and then superimposing the position and size of the basketballin each of the multiple images onto one of the images in the series. Thecomposite image 100 include, images of basketball (e.g., 120 and 122), abasketball hoop 106, net 108, backboard 102, support structure 104,trees (110 and 112), court 114 and edge of the court 130. The basketballis being shot towards the hoop by a person 116. The line 132 representswhere the court 114 is cut off in the image. In this example, the imageincludes trees because the basketball court 114 is outdoors. In otherembodiments, images can be captured inside.

In one embodiment, a digital camera on a smart phone can be used tocapture the 2-D images used to generate a composite image. The cameracan be fixed in placed using a number of methods, such as mounting thesmart phone to a flexible tripod (e.g., Joby Gorillapod™, www.Joby.com)which rests upon or is secured to a surface, leaning the smart phoneagainst an object, such as a fence, or holding the smart phone againstan object, such as a fence. Other image capture devices can be used asmart phone camera is provided for illustrative purposes only. Forexample, a tablet computer with an image capture device can be utilized.

In particular embodiments, the 2-D image data from the camera can beprovided as a two dimensional array of pixels where a number of valuesare associated with each pixel and where two pixel coordinates identifya position of each pixel the 2-D array. The size of the pixel array canvary from camera to camera depending on the sensor on the camera and thecamera settings. In each of the captured 2-D images, object recognitionsoftware can be used to detect the presence of the basketball anddetermine a pixel position and a size of the basketball in pixels ineach image by using the values associated with each pixel.

To generate the representative composite image 100, one of the series ofimages captured from the video camera can be selected and a basketballwith a size and a position as determined from the other images can besuperimposed on the one selected 2-D image. If the time between imagesis small enough, the superimposed basketballs can overlap with oneanother. The determined positions of the basketball in pixel space canbe used to construct a trajectory in pixel space. For instance, as shownin FIG. 1 , a trajectory line 128 can be drawn through the determinedcenter of the basketballs (e.g., 126) in pixel space to portray atrajectory of the basketball. The trajectory line 128 can be specifiedin 2-D pixel coordinates.

To generate a 3-D trajectory in physical space, which can be referred toas true space, an orientation of the camera, i.e., the direction thelens is pointing can be determined using accelerometers in the portableelectronic device (e.g., a smart phone). In one embodiment, as isdiscussed in more detail below, using the accelerometer data, areference plane relative to the center of the Earth and orientation ofthe camera relative to the reference plane can be determined. Thecaptured 2-D image data can be projected onto the determined referenceplane in pixel space. Next, a known size of the basketball can be usedto relate a length in pixels to a length in physical space to determinethe position of the basketball on the reference plane in true space,i.e., 3-D physical space.

In other embodiments, the orientation of a camera can be measured byhand and input. For example, a tripod can include rotation joints withangle markings which allow an orientation of the camera to be determinedrelative to some surface, such as the ground. As another example, acamera on a platform with rotation joints where the rotation jointsinclude markings which allow an orientation of the camera to bedetermined relative to some surface, such as the ground or a verticalwall, can be used to determine an orientation of the camera. In anotherexample, a user may be able to make measurement with a tape measure,ruler or some other measuring device which allows an orientation of thecamera relative to some surface to be determined. These measurements canbe input into the system.

In yet other embodiments, the camera position can be assumed to be acertain orientation based upon a specified installation orientation. Forexample, the camera lens can be included in a device which is coupled toa wall, such that the camera lens is parallel to the wall. Further, thedevice can be coupled to the wall, such that the bottom of images froman image sensor associated with the camera are parallel to a horizontalsurface, such as the floor.

As shown in the 2-D composite image 100, the size of the ball in theimage varies along the trajectory line 128. In particular, the size ofthe basketball 122 at the beginning of the trajectory appears smallerthan the size of the basketball 120 near the hoop 106. A size of thebasketball 122 near the beginning of the trajectory is shown within thesize of the basketball 120 to illustrate the change in size.

The size changes because the basketball is farther away from the imagecapture device at the beginning of the trajectory and then moves closerto the camera. Thus, as the basketball moves closer to the camera itappears larger in the 2-D image data captured by the camera. For adifferent shot, with the camera at the same position, the basketball mayappear bigger at the beginning of the trajectory (i.e., when it is shot)and then get smaller as basketball approaches the hoop. Further, for theshot in FIG. 1 , a camera at a different position, such as onepositioned on the other side of the court 114, may generate images wherethe ball is larger at the beginning of the trajectory and becomessmaller as the ball approaches the hoop 106.

In general, objects appear differently in 2-D image data depending onthe position of the camera relative to the object. However, because asphere is symmetric in any direction, it approximately appears in 2-Dimage data as a circle from any direction in which it is viewed by thecamera as long as lens aberrations and blurring due to pixel size areconsidered. The size of the circle depends on the distance from thecamera to the sphere.

For other non-symmetric objects, this property doesn't hold. Forexample, a person can appear quite differently in 2-D image datadepending on the position relationship of the camera relative to theperson. However, for axisymmetric objects, such as an American football,rugby ball, Australian's rules football, Frisbee or discus, it may bepossible to identify at least one characteristic length associated withthe object from any orientation of the object relative to a camera.Using the physical dimensions of the one characteristic length, it maybe possible to apply the methods and apparatus described herein todetermine the trajectory of the object in 3-D. Thus, the method andapparatus is not limited to spherical objects. A few examples ofnon-spherical objects axis-symmetric objects are described with respectto FIGS. 8A, 8B and 8C.

In general, the method and apparatus described herein can be appliedwhen a characteristic length associated with an object can be identifiedin image data from orientations relative to the camera in which theobject is expected to appear. For example, knowing the shape anddimensions of a tennis racket or a paddle, it may be possible toidentify in captured image data a characteristic length of the tennisracket, such as a diameter of the face, from most if not all of theorientations of the tennis racket relative to the camera. Based on theidentified characteristic length, its size in the image data and itsknown physical dimensions, it may be possible its distance to the imagecapture device and place its location in 3-D physical space, as isdiscussed more detail below.

To determine a distance to a spherical object, a physical dimension ofthe object can be used. For example, for a basketball, its diametermeasured in some length unit, can be used. In one embodiment, the typeof ball, such as a men's ball or a women's ball, or the diameter of theball can be input by a user. With the dimensions of the ball specified,a distance from the camera to the ball can be determined using 2-D imagedata. Using the determined distance from the camera to the ball, adetermined position of the camera based upon accelerometer data and 2-Dimage data captured from the camera, a position of an object in 3-Dphysical space can be constructed.

In more detail, the images in which the basketball appears can be takenat different times where the time between images can be determined.Thus, the position of the basketball as a function of time in 3-Dphysical coordinates can be determined. Using the position of the ballas a function of time, other quantities can be derived, such as a 3-Dvelocity vector for the ball and a speed of the ball at different pointsalong its trajectory based upon its change in position from image toimage.

When a series of images is generated, such as via a video camera, all ofimages in the series don't have to be utilized. For example, if thevideo camera is recording images at 30 frames/sec and video is takenover some time period, all the frames, every other frame, every thirdframe, etc. can be utilized. Further, if an object of interest, such asa ball, is obscured an image or the ball can't be identified in a videoframe, then the video frame may not be utilized.

In the image 100, basketballs are shown starting at a release height 118above the court 114 and following along a trajectory up to the hoop 106.In general, the motion of a ball in 3-D at any time can be captured andcharacterized as long as it appears in the 2-D image data. For example,a motion of the ball can be characterized while a person is holding thebasketball ball if it is at least partially visible in the captured 2-Dimage data.

In basketball, a person may hold the ball while driving to the basket orwhile catching a pass and then initiating a shot. The basketball may beonly partially visible at times in the 2-D image data because parts ofthe ball can be obscured by a person's hand or body. Nevertheless, thesystem can be configured to recognize the ball in the image data andidentify its position in view of its known circular appearance in theimage data. In addition, the motion of ball can be characterized afterstriking the backboard or the rim, while it is being dribbled, while itis being passed or while it rolls along a surface, such as rollingaround the rim or rolling on the ground. In general, a trajectory ofball can be determined while it is in flight, in contact with a surfaceor being carried in some manner.

If a ball is moving with a person, then a motion of a ball can also beused to characterize a motion of the person. For example, if a soccerplayer is dribbling a soccer ball near their feet, than the rate theball is moving is similar to the rate the person is moving. Thus, therate of movement of the person dribbling the soccer ball can becharacterized. As another example, if a lacrosse player is running whileholding the ball, then the velocity of the ball characterizes thevelocity at which person is running.

In the examples above, the 3-D position and velocity of a ball inphysical space can be determined using 2-D image data received from animage capture device, such as a camera, and knowing the size of theball. When other secondary objects can be identified in the image datawhich have a known size, a distance of the camera to the secondaryobject and a 3-D position of the secondary object in physical space canbe determined. With this information and position of the ball in 3-Dphysical space, a position of the ball relative to the secondary objectcan be determined.

As an example, in FIG. 1 , the distance of the camera to the hoop 106can be determined. If the hoop is standard hoop at a standard height,the hoop dimensions and the height of the hoop are known. Thus, adistance of the hoop to the camera can be determined and a position ofthe hoop in the same physical coordinate system as the ball can bedetermined. Then, a position of the ball relative to the hoop and otherparameters, such as an entry angle of the ball as it approaches the hoopcan be determined.

In other example, it may be possible to identify markings/boundaries ona court which have a known size and position, such asmarkings/boundaries on a basketball court, markings/boundaries on atennis court or marking/boundaries on a bowling lane. Using thisinformation, it may be possible to determine the coordinates of thebasketball court, tennis court or bowling alley in 3-D physical spacerelative to the camera. Also, it may be possible to identify an objectwith known dimensions to provide additional position references. Forexample, a bowling pin, a base/home plate in baseball or a soccer goalin soccer, which each have known standard sizes, can be identified in2-D image data and positions of these objects in a 3-D physical spacedefined relative to the camera can be determined. Then, a ball can beidentified in 2-D image data from the camera and mapped to the same 3-Dphysical space which allows the position of the ball relative to theobject to be determined.

The accuracy, with which a ball's trajectory is characterized, such as abasketball, can depend on the resolution of the camera, the distance ofthe camera from the object and any magnifying optics which are employed(e.g., a zoom lens). The ball will appear in a number of pixels wherethe number of pixels depends on the camera resolution and the distanceof the camera from the ball. In general, as the number of pixels inwhich the ball appears increases, the ability to resolve the edges ofthe basketball and accurately determine its diameter in pixelsincreases. The accuracy of the diameter of the ball in pixels affectsthe accuracy at which the distance from the ball to the camera inphysical space is determined.

In some embodiments, the ball may move a distance away from the cameraduring a series of frames such that the distance to the ball may nolonger be accurately determined. For example, a smart phone camera usedby a person sitting behind home plate of a baseball diamond may be ableto determine a distance to the ball after it is hit for its entiretrajectory if it is hit in the infield. However, if the ball is ahomerun, as it approaches the outfield fence, it may not be possible toaccurately track the distance to the ball. As another example, for smartphone camera behind a golf ball being hit off the tee, after the golfball travels some distance, a camera resolution may not be able toaccurately determine a distance to the golf ball.

In one embodiment, the system can be configured to cut-off tracking atsome threshold value. For example, when the distance associated with apixel divided by a diameter of the ball is above some percentage, suchas 10% or 25%, the system may no longer attempt to track the trajectoryof the ball and/or may indicate the trajectory data after some point inthe trajectory where the threshold is surpassed is not accurate. Forexample, a trajectory of baseball which is determined can be output ingreen, yellow and red where green indicates sufficient accuracy, yellowindicates marginal accuracy and red indicates a relatively inaccuratecalculation.

In another embodiment, the ball characterization methodology using 2-Dimage data can be combined with or enhanced by a predicted methodology.In particular, when the trajectory of an object can no longer beaccurately determined using image data, the physical equations of motionfor the object can be solved to extrapolate the 3-D trajectory of a balldetermined using the 2-D image data. For example, a first portion of ahome run shot or a golf shot in 3-D physical space can be determinedusing the 2-D image data, then a second portion can be estimated using apredictive methodology which solves the physical equations of motion forthe object.

When the physical equations are solved, the initial conditions for thepredictive methodology, such as an initial position of ball and avelocity vector for the ball in physical space can be supplied form the3-D trajectory determined via the 2-D image data. For example, for abaseball trajectory, the predictive methodology can be applied after aball leaves the infield where the initial conditions for the predictivemethodology are the position and velocity of the ball as it leaves theinfield. As another example, for a golf trajectory where the camera isplaced near where the ball is hit, the predictive methodology can beapplied after the ball has travelled some distance (e.g., 100 or 150yards). In yet another example, for a golf trajectory where the camerais placed where the ball is hit, the predictive methodology can beapplied starting at a point just before the ball becomes obscured by anobject, such as passing behind trees or a hill which obscures a view ofthe ball.

In one embodiment, trajectories determined using the 2-D image data anda predictive methodology can be output in a side by side manner. Forexample, the initial conditions of a golf shot being hit can bedetermined from 2-D image data and the trajectory can be determined in3-D using the 2-D image data. The predicted trajectory can be generatedbased upon the initial conditions near the ball being hit and thepredicted trajectory can be plotted next to the actual 3-D trajectorydetermined from the image data.

In yet another embodiment, a number of trajectories can be generatedfrom a single set of initial conditions. For example, for a hitbaseball, which is a homerun, factors such as wind speed, temperatureand ball spin can be varied through a range to predict a number ofpossible trajectories based upon 3-D initial conditions determined fromthe 2-D image data (e.g., the initial conditions may start after theball has reach the outfield and 3-D trajectory determined from the 2-Dimage data becomes less accurate). These simulations can generate alanding ellipse in which the ball is predicted land. It is similar to acone that is generated for predicting a path of a hurricane from acertain point. The landing ellipse can be used to provide a range ofdistances from home plate at which it is believed the homerun ball wouldhave landed if it had not landed in the stands. Additional details ofthese predictive methodologies, which can be utilized herein, aredescribed in is related to U.S. patent application Ser. No. 12/127,744,filed May 27, 2008, by Marty, et al, title, “Stereoscopic Image Capturewith Performance Outcome Prediction in Sporting Environments,” which isincorporated in its entirety and for all purposes.

FIG. 2 is a representation of a 2-D composite image 200 including atrajectory of baseball in flight in accordance with the describedembodiments. A camera is positioned in front of an individual throwingthe baseball. For example, the camera, which may be a smart phone, maybe positioned behind a home plate and a catcher to which the person 202is throwing. The person 202 in the image is shown in a position afterreleasing the baseball.

Baseball motions from other types of throws can be captured andcharacterized from 2-D image data, such as between players. In addition,motions of a hit ball can be characterized. Thus, the example of aperson pitching is provided for the purposes of illustration only and isnot meant to be limiting.

Unlike a basketball, which primarily travels in a parabolic/planartrajectory after release, a baseball's trajectory can be much more threedimensional. As a result of spin placed upon the ball, the ball maysink, rise or move side-to-side. Using the 2-D image data from thecamera and a known size of the baseball a trajectory of the baseball in3-D physical space can be determined.

In Figure w, the baseball is shown at a starting position 204, near itsrelease, and an ending position 206, before it is caught. The baseballcan be identified in a series of 2-D images. Then, as shown in FIG. 2 ,a line 210 can be drawn through the centers (e.g., 208) of thebaseballs. Using the image data, the positions of the centers of thebaseball can be determined in both 2-D pixel coordinates and 3-Dphysical coordinates.

The baseball appears to increase in size after it is released. The sizeincreases because the baseball is getting closer to the camera. Theinitial size 204 of the baseball relative to the final size 206 isshown. In FIG. 2 , the relative change in size is much greater than inFIG. 1 where the ball is moving more parallel to the camera.

As described above, 2-D image data captured from a digital camera can beused to determine a three dimensional trajectory of a ball. Thus, somedetails of a digital camera are described as follows. FIG. 3 is a systemdiagram 300 of a digital camera which generates 2-D images in accordancewith the described embodiments.

In FIG. 3 , a field of view 302 of the camera which shows a person afterhaving thrown a ball is shown. Light 304 from the field of view can passthrough one or more lenses and then captured by a sensor. Typical, adigital camera will use one of a charge-coupled device (CCD) sensor 302or a complementary metal oxides sensor (CMOS) 304. When light hits a CCDsensor 302 at a location, a current is generated. The current can beconverted to a digital value using an analog digital converter, such as306. In a CMOS 304, light hitting a sensor also generates a charge.However, an analog to digital converter is built into each pixel toconvert the measured charge into a digital value.

Output from the CCD sensor 302 and Analog to digital converter 306 orthe CMOS 304 can be array of digital values where each element in thearray can be referred to as a pixel. Multiple values based upon colorfiltering can be generated for each pixel from the sensor. These valuescan be referred to as sensor counts.

The output from the sensor can have noise. Noise may result fromdefective pixels, such as “hot” pixels which are always on or “dead”pixels which record nothing. Other sources of noise are also possible.The noise reduction 308 attempts to reduce this noise. For example, deadpixels can be masked and a value can be determined as an average fromsurrounding pixels.

The output from the noise reduction 308 can be a Bayer pattern. TheBayer patterns include patterns of color resulting from a Bayer filterplaced over the sensor. The Bayer patterns are assembled to create acolor image. The Bayer patterns resemble a tile mosaic. In 312, thepatterns are input to a demosaicing algorithm to form an RGB image.

Prior to demosaicing, the raw data can be stored in 310. One example ofa file type is referred to as a digital negative (DNG) file. In someembodiments, the raw data can be processed in a manner that isbeneficial to object recognition but not necessarily good for generatinga recognizable RGB image. In various embodiments, object recognition canbe applied to image data before and/or after demosaicing.

In 314, tone mapping can be applied. Tone mapping adjusts the perceivedbrightness of image pixels but doesn't change the color balance. Colorbalancing can be performed using a white point adjustment. Gammacorrection can be performed to affect the perceived brightness of theimage. Image tuning 316 can involve applying the different algorithms toaffect the output of image, such as sharpening or blurring edges. Thefinal image is output in 318 and can be stored.

Object identification 320 attempts to identify the ball including itsdimensions and location in 2-D pixel coordinates. It can operate on thefinal image from 318 or data in a rawer format. For example, in someinstances, special demosaicing algorithms can be applied to the raw data310, which are beneficial for object recognition, such as edgedetection.

Object identification 320 can also involve attempting to identifysecondary objects, such as a hoop used in basketball, a pin in bowling,a net in tennis or a goal in soccer. If the type of sport is identifiedas input, then the system can be configured to look for particularobjects associated with the sport, such as the hoop in basketball or abase/home plate in baseball. The object identification can involveutilizing known shapes associated with an object in a particular sport.

Object coordinate determination 322 can refer to determining thepositions of a ball in true space (3-D physical coordinates) using the2-D image data. In addition, the positions of secondary objects and theposition of the ball relative to the secondary objects can also bedetermined. Details of one methodology for performing this determinationare described as follows with respect to FIGS. 4A, 4B and 5 .

FIG. 4A is an illustration of a digital camera setup 400 in accordancewith the described embodiments. In FIG. 1 , the game reference frame 404can be relative to some surface, indoors or outdoors where a ball isbeing launched and tracked. For example, in 404, the X and Y directionscan be in a horizontal plane which is a playing field for the sport, andthe Z direction can be a distance above the horizontal plane. Forexample, the playing field can be an indoor or outdoor baseball diamond,basketball court or tennis court. Often the playing field is level, butin some instances, such as golf, the playing field can be some outdoorterrain, which can vary in height.

In one embodiment, a mobile computing device, such as a smart phone canbe positioned to relative to the game reference frame. The smart phonecan include a processor, a memory, a display, a network interface and avideo camera configured to capture a series of video images including atime associated with each image. An application for performing themethods described herein can be downloaded to the phone from a remotecode repository. The 2-D image data captured using the smart phone'scamera can be used to generate trajectory information associated with aball.

Information about trajectories and/or the 2-D image data can be outputvia the phone's display and/or other output mechanisms and/or uploadedto remote servers, such as a social media web-site, for sharing withother users and/or storage if desired. In addition, raw data used togenerate the trajectory information can be stored on the phone oruploaded to remote servers. Further, the trajectory information can becombined with the 2-D image data. For example, a composite image asdescribed above with respect to FIGS. 1 and 2 can be generated where atrajectory of the object is depicted.

In FIG. 4A, a camera reference frame 408 is shown. In the camerareference frame, one axis is aligned with a line perpendicular to thecamera lens. Using an accelerometer on the phone, the camera referenceframe 406 can be related to an Earth reference frame 406. Therelationship between the camera frame and Earth reference frame isindicated as yaw 410 a, roll 410 b and tilt/pitch 410 c values.Typically, at least 2 of the three of yaw, roll and pitch are availablefrom a smart phone's accelerometer. A position vector 412 relating theEarth reference frame 406 to the game reference frame 404 is shown.

The combination of yaw-roll-tilt information from the smart phoneaccelerometers and the resolution information from the camera enables aboard processor to relate the 2-D pixel arrangement in the camera fieldof view to the 3-D reference frame in the real world. In one embodiment,as referred to below as the camera accelerometer calibration, the 2-Dpixel data for each picture can be translated to a reference frame as ifthe camera where resting on a horizontal plane perpendicular to an axisthrough the gravitational center of the earth where a line drawn throughthe center of lens perpendicular to the surface of lens is mapped to acenter of the pixel data (e.g., see FIG. 4B). Since the ball isspherical, it appears the same (round in the image) if the camera isrotated around the axis through the gravitational center of the earth,i.e., the field of view is changed, as long as the object is in thepicture. For non-spherical objects, this relationship doesn't hold.

FIG. 4B is an illustration 450 showing a 3-D object projected onto a 2-Dimage captured by a digital camera in accordance with the describedembodiments. In 450, two images from a camera at a fixed position acamera are shown. The first image includes a first ball 460 and thesecond image includes a second ball 458. As described above, the 2-Dimage data is translated to a reference frame as is if the camera isresting on a horizontal plane perpendicular to an axis through thegravitational center of the earth where a line 456 c is drawn throughthe center of the lens 452 is mapped to a center of the pixel data.

A ball, such as 460 and 458 can be identified in each 2-D image usingobject identification methods, such as edge detection. In oneembodiment, the diameter in pixels which is measured is the diameterperpendicular to the flight line. This perpendicular diameter is usuallythe smallest diameter of the sphere since there can be motion smearingalong the diameter parallel to the flight line. Occasionally, though,another diameter will be smaller due to light blooming. Light bloomingis the result of scattering in the lens, which is interpreted as abright spot in a scene. For example, a bright light in the backgroundwill appear to bleed over onto objects in the foreground.

In particular embodiments, multiple diameters of a ball identified inthe pixel data can be determined in different directions and thenaveraged. The average value can be used in the distance determination.Besides the diameter, a center of the ball can be determined in pixelcoordinates.

Based upon a known size of the ball and the number of pixels associatedwith a diameter of the ball, a physical distance of the ball to thecamera can be determined. The position of balls 460 and 458 on the imagedata in physical space are indicated by spheres 464 and 462,respectively. A physical distance to the balls, 464 and 462, isindicated by the length of lines 466 and 468. Using the determinedphysical distance to the ball, pixel coordinates in the x, 456 a, and y,456 b, and the angles from the indicated geometry, x, y coordinatevalues in the pixel plane can be converted to x, y coordinates inphysical space. In addition, a physical distance perpendicular to the x,y plane in physical space along line 456 c can be determined. Thus, theposition of the ball in 3-D physical space along its trajectory can bedetermined based upon its position in each of the 2-D images which arecaptured. Further, the location of the camera (center of the lens) inthe same reference frame can be also be determined as shown in FIG. 4A.

For each of a series of 2-D images, the x, y, z position of the ball inphysical space can be determined. The time at which image is generatedis generally known for each image. Typically, a camera provides a timestamp for each image. Further, for a video camera, the frame rate, whichis usually expressed as video frames captured per second is a knownquantity for a camera. With this information, the x, y, z position ofthe ball in physical space as a function of time can be determined.

With the x, y, z position of the ball known as a function of timedetermined, a velocity and an acceleration of the object in eachdirection can be generated at each time by determining the appropriatederivatives. Then, information related to position, direction, velocityand acceleration at one or more positions of the ball captured in theimage data can be output to the user immediately or at some later time.For example, for a baseball pitch, a trajectory of the baseball can beoutput to a display device on the camera device, a speed associated withthe pitch can be output to the display or in audio format. Further, ifthe position of the plate has been determined in 3-D space, it may bepossible to determine a position of the ball relative to the plate.

Additional details of two methods for determining a 3-D trajectory of aball using 2-D image data are described as follows with respect to FIGS.5-8 . Prior to determining a 3-D trajectory, a number of calibrationsteps can be performed which can be used for one or both methods aredescribed. The calibration steps are described with respect to FIG. 5 .Then, a first method of determining the 3-D trajectory of a ball isdescribed with respect to FIG. 6 and a second method of determining the3-D trajectory of a ball is described with respect to FIGS. 7 and 8 .

FIG. 5 is a flow chart 502 including one or more calibration steps whichcan be used to determine a trajectory of an object with an identifiablecharacteristic length, such as an axisymmetric object, in 3-D physicalspace using 2-D image data. In 502, lens distortion parameters can bedetermined. When a checker board pattern of equally sized squares isheld up to the camera, an image with equally sized squares is not alwaysgenerated as a result of lens aberrations. For example, some of thesquares can be distorted and of different sizes than one another. In502, lens distortion parameters can be determined. In 504, a lenscalibration can be developed which adjusts the pixel sizes in the 2-Dimage data such that a 2-D image of a pattern of squares of equal sizesis reproduced in the 2-D image data. This calibration can be appliedeach time a 2-D image is captured.

In 506, the acceleration calibration can determined based uponaccelerometer data associated with the position of the camera (i.e., theposition of the mobile computing device which includes the camera). Asdescribed above, the pixel data for each captured image can betranslated to a reference frame as if the camera where resting on ahorizontal plane perpendicular to an axis through the gravitationalcenter of the earth where a line drawn through the center of lensperpendicular to the surface of lens is mapped to a center of the pixeldata (e.g., see FIG. 4B). The accelerometer calibration can be appliedto the pixels in each of the 2-D images which are analyzed.

When the position of the camera is fixed, the accelerometer calibrationis essentially the same for each captured 2-D image. In one embodiment,it may be possible to receive accelerometer data and match it on a frameby frame basis to each 2-D image. When a camera is being held andslightly moving, the accelerometer calibration can be determined on aframe by frame basis using the received accelerometer data. Thus, it maybe possible to generate a 3-D trajectory for a ball from images obtainedfrom a video camera which is held and moving to some degree as opposedto being fixed in a position relative to a surface. If a stationaryobject of a known size is present in each of the 2-D images, such asbasketball hoop, it may be possible to continually determine theorientation and distance of the camera to the object and thus, constructa 3-D trajectory of ball from 2-D video images taken from a movingdigital camera.

In 508, a distance calibration can be determined. Based upon a knownsize of an object and a resolution of the video camera, tables, whichrelate a determined pixel length of the diameter of a ball or somecharacteristic length of a ball, to a distance to the ball from thecamera can be generated. For example, for a baseball captured in a 2-Dimage by a camera of a specified resolution, a physical distance to theball from the camera can be determined from the table by specifying thediameter of the ball in pixels which is detected in the image. In otherembodiments, the determination of the physical distance of the ball tothe camera can be done on the fly without using the distancecalibration. In some embodiments, a user can select a resolution ofimages to be captured and then a selected resolution can be provided asinput.

In 510, direction calibration values can be determined. As describedabove, the 2-D image data can be transformed such that it corresponds toa camera aligned with a center of the 2-D image data (i.e., as if theimage data is generated by a camera lens parallel to the 2-D image planewhere a line passing through the center of the 2-D image data passesthrough and is perpendicular to a center of the camera lens.) An anglecan be defined between a first line drawn through a center of the imagedata and center of the camera lens and a second line drawn from thecenter of the camera lens to a pixel location on the 2-D image plane(see FIG. 4B). The value of the angle at each point in the pixel arraydepends on the distance from the camera lens to the ball. Once thedistance between the ball and the camera is determined, the angle isdetermined.

In 510, tables can be generated which relate physical distance to anglefor a number of different distances. Thus, if a distance and pixelcoordinates are specified, the angle at the pixel coordinate can belooked up from the table. Interpolation and extrapolation schemes can beutilized when the angles for a particular distance have not been storedto the table, such as linear or higher order extrapolation. Once theangle is known, the position of the ball in 3-D physical space can bespecified. As described above for the accelerometer calibration and thedistance calibration, the angle determination can be done on the flywithout the use of a lookup table. However, the use of a lookup tablemay reduce a time needed to determine a 3-D trajectory of a ball.

Next details of a first method for determining a 3-D trajectory of aball from 2-D image data is described. FIG. 6 is a flow chart of amethod 600 for determining a 3-D trajectory of a spherical object using2-D image data in accordance with the described embodiments. In 602, 2-Dimages of ball in flight and/or rolling along a surface (e.g., a golfball rolling along the ground, a basketball rolling around a basketballhoop or a bowling ball rolling down a bowling lane) can be captured.Further, 2-D images of a ball in contact with a surface can be captured,such as a ball being held/moved by a body part of a person (e.g.,held/moved by a hand or foot), being struck by an object (e.g., aracquet held or a bat held by a person) or striking and bouncing off asurface (e.g., a basketball hoop or a playing surface). A singletrajectory can include multiple components, such as a ball being held,in flight and rolling along a surface, during its trajectory.

In 604, a lens calibration, as described above, with respect to FIG. 5 ,can be applied to first image data in a series of captured 2-D images.In 606, the accelerometer calibration can be applied to the first imagedata. The accelerometer calibration, described above with respect toFIG. 5 , accounts for one or more of yaw, roll, and tilt/pitch of thecamera and transforms the first image data to a reference frame as ifthe camera is resting on a horizontal plane perpendicular to an axisthrough the gravitational center of the earth where a line drawn throughthe center of lens perpendicular to the surface of lens is mapped to acenter of the pixel data. Other reference frames can be utilized andthis one is provided for the purposes of illustration only and is notmeant to be limiting.

In 608, using object recognition methods applied to pixel data, such asedge detection methods, a ball can be located in the first image dataand a diameter of the ball in pixels can be determined. In someembodiments, the diameter of the ball in pixels can be determined in oneor more directions. When the diameter is determined in multipledirections, in some embodiments, the multiple diameters can be averagedin some manner. In a particular embodiment, the determined number ofpixels of the diameter is below a threshold value, the system may notdetermine a distance to the object because of the error marginassociated with the calculation. The threshold value may depend on aresolution of the camera which is being utilized.

In 610, a position of the ball in 3-D physical space can be determinedusing the distance calibration and direction vector calibration with thefirst image data. As described above, the distance calibration canrelate the diameter of the ball in pixels to a distance to the ball fromthe camera. The direction vector calibration can be used to determineone or more angles associated with the distance to the ball from thecamera. In one embodiment, distance calibration value and directionvector calibration values can be determined using a table lookupprocedure. In another embodiment, the distance calibration values anddirection vector calibration values can be determined on the fly.

In 612, the system may attempt to locate one or more secondary objects.The identification of secondary objects is optional. In one embodiment,the type of ball and/or whether it is being played on a marked fieldusing standard equipment can be input into the system. For example, thesystem can receive input that a women's basketball is being used on amarked basketball hoop with a standard hoop height and standard hoop.Based upon this information, the system can be configured to attempt tolocate one or more, such as the hoop and/or one or more court markingsin the first image data.

As another example, the system can receive an input that the ball is astandard baseball and a game is being played on a little league baseballdiamond. Based upon this information, the system can be configured toattempt to locate one or more bases and construct positions of a littleleague baseball diamond relative to the camera in 3-D space using knowndimensions of a little league baseball diamond. Then, the trajectory ofthrown or hit balls can be determined and placed relative to thebaseball diamond.

In one embodiment, the system can be configured to allow a user toidentify dimensions for a size of a non-standard object. For example,the system can be configured to allow a user to receive an input of asize of a non-standard soccer goal. In one embodiment, the user can takea picture of an object, trace its outline and then enter its dimensionsor identify it some manner. For example, a picture including soccer goalcan be taken and the system can output an image of the goal to atouchscreen display. Then, the user can trace over a portion of theobject and/or draw a shape around the object. The received touch inputcan be used by the system to identify the object in the image and otherrelated images.

As another example, the system can output an image of a tennis court andthe user can trace over court lines or a top of a net on the tenniscourt. For example, the system can be configured to prompt the user totrace over the top of the net or identify it some other manner in theimage. The system can receive this input and use it to help identify thenet in the output image and other images.

As described above, the system can use the information received from thetouch screen to attempt to identify the object of interest highlightedby the user. Then, in one embodiment, the system can request dimensionsfrom the user for one or more portions of the object. The dimensioninformation can allow the system to determine a distance to thesecondary object using the 2-D image data. In addition, the system canbe configured to identify the secondary object in subsequent images. Inone embodiment, object identification of secondary objects can beperformed as part of a calibration procedure described above withrespect to FIG. 5 .

In 614, the system can attempt to identify the secondary object in theimage. Then, a length of some portion of the object with a known lengthin physical space can be determined in pixels. If an object has beenidentified in a previous 2-D image in a series of images and the objectis stationary and the camera is stationary, the system may attempt tolook for the object in approximately the same location at which it hasbeen previously identified.

Further, if a secondary object can't be identified because it isobscured in one of the images, the system can be configured to assumethe secondary object is at its previously detected location. Forexample, a camera view of a secondary object, such as soccer goal or abase in baseball can be obscured by the presence of a person between thecamera and the object, such a person standing on the base or in front ofthe goal, in one of a series of images including the object. However,based upon an identification of the object, in one of the other imagesin the series, the object can be assumed to be at the same location inthe image.

In one embodiment, a stationary object may only be identified once in aseries of images and then assumed to be in the same location for therest of the images. For example, a position of a basketball hoop can beidentified once and as long as the position of the camera is not changedas indicated by the accelerometer data, the system can configured toassume the position of the basketball hoop has not changed and is thesame in subsequent images. When the system, determines the position ofthe camera is adjusted, such as via a change in the accelerometer data,the system can be configured to attempt to re-identify the object in theimage data.

In one embodiment, a camera can be held such that the position of thecamera is changing over time in a series of images. In this case, thesystem can be configured to attempt to identify a secondary object, suchas a basketball hoop in each of the images. The basketball hoop canprovide a reference point with a known fixed position and physicaldimensions and then the trajectory of the basketball can be placedrelative to the basketball hoop in each of the series of images eventhough the camera may move from image to image.

In 616, using the distance calibration and direction vector calibration,a distance to one or more portions of the object can be determined inphysical space. In 618, if desired, a position of the ball relative tothe secondary object can be determined, such as an angle at which abasketball approaches a hoop or a location within in a goal at which asoccer ball crosses the plane of the ball.

In 612, when there are no additional secondary objects to be located,the system can determine in 620, whether there are additional images tobe processed. When there are additional images to be processed in asequence of images, the method can return to 606. When there are noadditional images, in 622, the trajectory in 3-D space can beconstructed. In one embodiment, as described above, analyticalpredictive methods can be used to determine portions of the trajectory.For example, physical equations of motion can be solved based uponinitial conditions determined from the trajectory determined from the2-D image data to extrapolate a 3-D trajectory beyond where thetrajectory can be accurately determined using the 2-D image data.

In 624, information associated with the determined 3-D trajectory can beoutput. For example, an image, such as shown in FIGS. 1 and 2 ,including one or more of the 3-D trajectories which have been generatedand/or portions of the 2-D images used to generate the trajectory (e.g.,one or more fixed objects from the images) can be output to a displaydevice. In addition, additional features can be added to output imagesthat are not necessarily captured in the images used to generate the 3-Dtrajectory. For example, if a portion of a tennis court including atennis ball trajectory is captured, the system can be configured torender in an image of an entire tennis court including markings andplace the captured 3-D trajectory to scale on the rendered tennis courtusing the portion of the tennis court captured in the image.

As another example, if a portion of a baseball field is captured and a3-D trajectory of a baseball is determined relative to the baseballfield, then an image including the 3-D trajectory an the entire baseballfield can be rendered and output. In yet another example, a 3-D baseballtrajectory determined on one field can be displayed on another baseballfield. For example, a 3-D homerun trajectory captured on one majorleague field can be rendered relative to another major league field todetermine whether it would still be a homerun on the other field.

In one embodiment, an interactive interface associated with the 3-Dtrajectory can be generated. The interactive interface can be configuredto allow a user to manipulate the trajectory, such as rotate it in spaceto allow it be viewed from different angles and see details about thetrajectory, such as velocities, accelerations and positions at differentpoints along the trajectory. For example, the speed of a baseball pitchat release or when it is crossing the plate can be determined and outputto a user.

Information about a 3-D trajectory and data used to generate the 3-Dtrajectory, such as 2-D image data can be stored to the mobile computingdevice, such as a smartphone. It can also be uploaded to a remoteserver. Further, in some instances, it can be shared with otherindividuals or posted to a social media site.

In one embodiment, statistical variations associated with a number oftrajectories and/or motions, such as body motions, captured in the 2-Dimage data, can be determined. The statistical variations can be used togenerate consistency parameters which measure how consistently atrajectory or a body motion is repeatedly generated. In one embodiment,the motion consistency parameters can be used as part of a video gameand/or as a basis for generating a training regimen for improvingconsistency of motion. Details of generating and using consistencyparameters are described in U.S. patent application Ser. No. 13/780,970,filed Feb. 28, 2013, by Marty, et al., which is incorporated byreference in its entirety and for all purposes.

Additional Trajectory Determination Methods

As described in this section, other methods involving the use of 2-Dimage data can be used in addition to or separate from ball sizedetermination to determine a position or an orientation of a ball in 3-Dspace as a function of time. As is described below, these methods mightnot be applicable under all conditions or for all objects. In oneembodiment, certain methods can be applied to parabolic and planartrajectories.

If forces, such as drag, lift, spin and wind, are minimal, an object,such as a spherical object, may follow a parabolic and planartrajectory. For example, a basketball indoors or outdoors without windessentially follows a parabolic and planar trajectory. As anotherexample, a shot put essential follows a planar and parabolic trajectory.

FIG. 7 is an illustration of a 2-D image of a parabolic and planartrajectory 704 for a basketball, which travels to hoop 710 in plane 702.The hoop 710 is mounted to a backboard 712. The start of the trajectoryis point 706 (release point). After the release, the basketball travelsin plane 702 with a vertical in-plane velocity and horizontal in-planevelocity, which vary as a function of time.

In one embodiment, a distance and orientation of an image capture devicecan be determined without using ball size. For example, a distance ofthe image capture device to a basketball hoop, can be determined fromidentifying the hoop and determining its diameter in pixels. With aphysical diameter of the hoop known, the physical distance of the camerato the hoop can be determined. For a standard hoop, the height of thehoop is also known.

Depending how the shape of the hoop appears in the image, an orientationof the image capture device relative to the hoop including its heightabove the ground can be determined. For example, if the camera is at thesame height as the hoop, the hoop will appear as a line. The hoop isassumed to be in a plane parallel to the ground. If the camera lens isresting on a plane parallel to the ground, the hoop will appear level inthe image. Else, the hoop will appear tilted in the image. Thus, theorientation of the hoop in image can be used to determine an orientationof the camera. In one embodiment, the determined orientation of theimage capture device relative to the hoop can be used to normalize theimage data relative to the ground (Earth reference frame) as wasdescribed above.

In alternate embodiments, the orientation of the horizontal and verticallines on the backboard 712 or the side and top edges of the backboard inthe image data can be used to determine an orientation of the camerarelative to the backboard. Further, an orientation of the hoop relativeto these lines or edges, alone or in combination, with a distance to thebackboard, can be used to determine an orientation of the camerarelative to the hoop and backboard. Again, when the orientation and/orposition of the camera are determined, it can be used to normalize the2-D image data.

In other sports, the appearance fixed objects with known dimensions canbe used to determine a position and orientation of a camera relative tothe object. For example, the appearance of football goal posts of knowndimensions in an image can be used to determine an orientation of thecamera and a distance of the camera to the goal posts. As anotherexample, the appearance of a soccer goal of known dimensions can be usedto determine a position and orientation of the camera relative to thegoal.

In the example above of trajectory 704, the position of the center ofthe hoop in 3-D space relative to the camera can be determined. If it isassumed the trajectory of the ball passes through the hoop, then theposition of the center of the hoop can be approximated as a point alongtrajectory 704. The location of point 706 might be partially determinedby knowing a release height of the shot. Most good shooters, such as NBAplayers, release the ball at a consistent height. Thus, if the shooteris identified, the height of point 706 can be assumed to be the releaseheight.

A position of a person on a court might be determined by identifyingcourt markings and a person's foot position relative to the courtmarkings can be used to determine a 3-D position of point 706 in 3-Dspace. If a time it takes a ball to travel from point 706 to the centerof the hoop can be determined, a velocity of the ball in the horizontalin-plane direction can be determined, since distance 708 is known. Inone embodiment, the time can be determined from the time between whenimages are captured. In the horizontal in-plane direction, the distancethe ball travels per time is constant. Thus, knowing the horizontal inplane velocity can be used to determine a physical position of the ballin the plane 702.

In the vertical in-plane direction, knowing that the ball rises andfalls according to the Earth's gravity, y position in physical space canbe determined which generates a trajectory 704 as it appears in theimage data. In particular, different trajectories can be generated andprojected onto the 2-D image until the trajectory matches the one in the2-D image. Knowing the orientation of plane 702 from the position of thecenter of the basket and the position of the release point 706, or ingeneral any two points which can be determined to be located in plane702, position as a function of time along trajectory 704 in plane 702(or in 3-D) can be estimated and properties of the trajectory can bedetermined.

Non-Spherical Objects

In this section, position determination for non-spherical objects using2-D image data is described. FIG. 8A is an illustration of a prolatespheroid shaped object in various orientations 800. A prolate spheroidis a spheroid in which the polar axis is greater than the equatorialdiameter. Prolate spheroids are elongated along a line, whereas oblatespheroids are contracted. An American football, a rugby ball and anAustralian rules football are approximately prolate spheroids.

In FIG. 8A, a cross section of the spheroid is ellipse 806. The ellipse806 has a major axis 808 a and a minor axis 808 b. The cross section indifferent orientations can be used to determine how the object willappear in 2-D image data taken by a camera.

A rotation 804 a around axis 802 a of the spheroid doesn't change thecross section in 2-D because the spheroid is symmetric around axis 804a. A rotation 804 b of spheroid around axis 802 b, which is through thecenter of the ellipse changes an orientation of the elliptical crosssection 806. A rotation 804 c of the spheroid around axis 802 c causesthe cross section to vary from elliptical cross section 806 to circularcross section 810. Depending on the amount of rotation around axis 802c, the cross section can appear as an ellipse of a varying size with amajor axis which varies in size between the diameter of circle 810 andthe length of the major axis of ellipse 806.

The cross sections associated with the various orientations 800 span howthe object can approximately appear in a 2-D image. In any of theorientations, the edge of the spheroid appears as an ellipse. For anyorientation and shape of the ellipses, the minor axis 808 b is alwaysthe same length where a circle is an ellipse where the major axis andminor axis are the same length. Therefore, if the length of the minoraxis can be identified in the image data and the physical length of theminor axis is known, the physical distance from the camera to the objectcan be determined from the image data as was described above.

Further, the ratio of the length of the major axis to the length minoraxis can provide some information about the orientation of the objectrelative to the camera. For example, when the ratio is one, the spheroidis being positioned end-on to the camera, i.e., the circular crosssection appears in the image data. When the ratio is a maximum value,the spheroid is being viewed from the side where it has its maximum areacross section. The orientation determination is not definite becausemultiple orientations can appear the same in the 2-D image data. Forexample, a football viewed end-on can look the same independent of whichend is pointing towards the camera. As another example, a ball rotatedthrough an angle around an axis in one direction can appear the samewhen it rotated through the same angle in the opposite direction.

In general, for any object with a variable position that has acharacteristic length that can be identified in 2-D image data for someset of orientations relative to a camera that are of interest, thedistance of the camera to the object can be determined in physical spacewhen the physical length of the characteristic length is known. In thecase of a sphere, the characteristic length that can be identified isthe diameter of the sphere. In the case of a football, thecharacteristic length is a minor axis of its elliptical cross section.In the case of a hoop, the characteristic length is the diameter of thehoop.

For the case of sphere, football or hoop, it is possible and/ordesirable to identify the characteristic length of the objects in 2-Dimage data for any orientation of the camera relative to the objectbecause depending on the trajectory parameters the object can appear.For other objects, it may not be necessary to identify the object fromany orientation. For instance, if the object is limited to a set oforientations during play, then it may not be necessary to identify thecharacteristic length from for the object from all orientations.

FIG. 8B is an illustration of a Frisbee shaped object in variousorientations 820. An edge-on cross section 822 has a diameter 824. Thediameter 824 can be considered the characteristic length. A rotation 804a of the Frisbee around axis 802 a doesn't change the shape the crosssection 822 which is viewed because the object is symmetric about thisaxis. A rotation 804 b of the cross section 822 around axis 802 bchanges the orientation of the cross section but not its shape. Arotation 804 c around axis 802 c causes the Frisbee to appear edge-onwith cross section 822 to an ellipse with major axis 824 up to a circlewith diameter 824.

Thus, if cross section 822 can be identified in the 2-D image data orone of the ellipses with major axis 824 can be identified in the 2-Dimage data and the characteristic length 824 can be determined, thephysical distance to the object to the camera can be determined when thephysical length of the major axis is known. Knowing the physicaldistance to the object and the orientation of the camera, as describedabove, the position of the object in 3-D space can be determined.Knowing the position as a function of time, such as from the times whenthe images were captured, trajectory parameters, such as a velocityvector, can be determined at various positions along the trajectory.

FIG. 8C is a representation of a 2-D composite image 850 including atrajectory 860 of a football in flight. To the camera, the outlines ofthe football appear approximately as ellipses of different sizes anddifferent orientations, such as 852 a, 852 b, 852 c and 852 b. For eachellipse, a center of the ellipses, such as 856 a, 856 b, 856 c and 856d, can be determined. Further, a minor axis of the ellipses, such as 854a, 854 b, 854 c and 854 d, can be determined. The distance to thefootball to the camera can be determined when the equatorial diameter ofthe football is known. The 3-D position of the football at each positioncan be determined using the methods described above, such as withrespect to FIGS. 5 and 6 . When the points are determined, the pointscan be connected to generate a trajectory, such as trajectory 860. Whenthe points are known as a function of time, parameters, such asvelocity, can be determined.

System

FIG. 9 is a block diagram of a system 900 including true space trackingof axisymmetric object flight (or object with an identifiablecharacteristic length) using 2-D image data in accordance with thedescribed embodiments. The components described with respect to FIG. 9can be implemented on a single electronic device, such as a smart phoneor divided between multiple devices. Each electronic device can includeone or more processors, a memory and network interfaces, such aswireless or wired communication interfaces. When the components aredivided between multiple devices, the network interfaces can allowcommunication between the devices.

In further detail, in one embodiment, a camera, sensors and processorcan be located on a mobile computing device where the mobile computingdevice is used to perform the trajectory analysis. In anotherembodiment, a camera and sensors can be located in a first device andthe trajectory determination can be performed on a separate device. Forexample, a first device including a camera and sensors can be mounted toa wall or a tripod where the first device can be configured tocommunicate image and/or sensor data to a second device. The seconddevice can be configured to perform the trajectory determinationincluding the object identification in the image data.

In yet another embodiment, a camera and/or sensor device can be worn.For example, a person can were glasses including a camera. The cameracan transmit images to a second device, such as a mobile phone, wherethe mobile phone performs the object identification and/or trajectorydetermination.

The captured 3-D trajectory data 902 may be used to perform trajectoryanalysis 122. The trajectory analysis 122 may be used to generateparameters which characterize a particular trajectory. For example, forgolf, a velocity and direction at which a golf ball leaves a club afterbeing struck with a club can be determined, for tennis, a velocity anddirection at which the tennis ball leaves the racket after being struckwith the racquet can be determined and for basketball, a velocity anddirection at which the ball leaves a shooters hand can be determined.

The trajectory capture 902 may only provide data related to a portion ofa trajectory, such as a beginning, middle or end portion of atrajectory. Using a trajectory analysis methodology, other portions of atrajectory which are not characterized can be simulated. In particular,after an initial portion of a trajectory is captured, a subsequentportion of the trajectory may be predicted. For instance, data regardinga player shooting a basketball may be used to predict whether the shotgoes through a hoop or not or how far a home run ball would travel. Inanother example, data regarding a golfer striking a golf ball may beused to predict a location where the golf ball will land, such as on agreen, when the 2-D image data doesn't have sufficient resolution toallow the golf ball to be tracked to the end of its trajectory or theend of its trajectory is obscured in some manner, such as from trees orhills.

Method and apparatus related to trajectory predictions may be referredto as outcome prediction 906. As another example, based upon images of abasketball approaching a basketball hoop, it may be possible to predictthe velocity, direction and angle of the ball as it left the shootershand. Thus, the beginning of a trajectory is predicted based on imagedata captured near the end of the trajectory.

Trajectory analysis 904 and outcome prediction 906 may be used as partof training methodologies that help a player to develop consistency inreproducing the factors that result in a particular trajectory of a balland thus, improve their skill level. Developing correct muscle memory isa term that is often associated with training methodologies that help aplayer develop a skill in a sport. The method and apparatus describedherein may be used to provide feedback information as part of trainingfeedback methods 920 that help a player develop consistent muscle memoryfor an activity associated with a particular sport.

Data related to trajectory capture 902 and trajectory analysis 906 maybe stored and archived 908 and later utilized for a number of differentpurposes. These purposes may include but are not limited to a)simulating trajectories 912 including utilizing Monte Carlo methods 910and b) scenario simulation 914 which may include applications related to2-D and 3-D rendered games 914. In one embodiment, the simulatedtrajectory analysis 912 may be used to quantify and determine optimumtrajectory factors for a particular trajectory, such as the best angleat which to launch a shot of a basketball.

The trajectory analysis 904, archived data 908, simulated trajectoryanalysis 912 and scenario simulation 914 may be used to evaluate theperformance 918 of individual participants or groups of participants.The evaluation of the performance may comprise quantifying aparticipant's ability to reproduce the factors related to generating aparticular trajectory. Once a player's ability is quantified in somemanner, the evaluation of performance 918 may include comparing 1) aparticipants performance against himself, such as past performanceagainst a current performance, 2) comparing a participant's performanceagainst another participants performance, 3) comparing a participant'sperformance against a defined standard, such as placing theparticipant's performance within a defined skill ranking and 4)comparing a participant's to some optimum, such as a comparing averagesof factors that a player produces to generate a particular trajectoryagainst optimum values of these factors determined from a simulatedtrajectory analysis 912.

In one embodiment, the performance evaluation 918 may includepredictions of future performance, such as an improvement in performancethat an individual might make if the individual were to change someaspect in the manner in which they generate the factors that produce aparticular trajectory or were to improve a consistency in which theygenerate the factors that produce a particular trajectory. This type ofanalysis might be performed using the simulated trajectory analysis 912including the Monte Carlo methods 912. In another embodiment, theperformance evaluation 918 may include a prediction of futureperformance, such as a win differential, that a group may make if theindividuals in the group were to change some aspect in the manner inwhich they generate the factors that produce a particular trajectory orwere to improve a consistency in which they generate the factors thatproduce a particular trajectory, such as if a basketball team improvedthe consistency at which they generated free throws. This type ofprediction may include scenario simulation 914.

In addition, a performance evaluation may be developed for a “composite”participant. For example, in basketball, the consistency at which agroup of participants generate factors that produce shot against aparticular defender may be determined. The shots made by each playeragainst the defender may be treated as if a single player had made eachof the shots and analyzed accordingly. In another example, theconsistency at which a group of participants in golf on a particulargolf hole generate a shot may be determined. The shots made by the groupof participants may be treated as if a single player had made each ofthe shots on the selected hole. The evaluation of performance for acomposite player may involve similar comparisons as described in theprevious paragraph for an individual player

Once performance is evaluated for an individual player, group of playersor a composite player, feedback information 922 may be provided. In manyinstances, the feedback information may be provided in a graphicalformat where the graphical format provides some indication of a levelconsistency at which the factors that produce a particular trajectoryare being generated. In a particular embodiment, the feedbackinformation 922 may be utilized in a real-time sports environment, suchas during a televised sporting event.

Methods and apparatus related to outcome prediction 906, simulatedtrajectory analysis 912 including Monte Carlo methods, performanceevaluation 918, feedback information 922 including broadcasting 924 andscenario simulation 914 including games 916 are can be utilized with the3-D trajectory determination described herein. Additional detailsregarding these elements are also described with respect to US.application Ser. No. 11/507,886, filed Aug. 21, 2006 and titled,“TRAJECTORY DETECTION AND FEEDBACK SYSTEM,” and U.S. application Ser.No. 13/745,429, titled “TRAJECTORY DETECTION AND FEEDBACK SYSTEM FORTENNIS,” filed Jan. 18, 2013, by Marty, et al., each of which isincorporated by reference in its entirety and for all purposes.

Embodiments of the present invention further relate to computer readablemedia that include executable program instructions for performingrecruiting techniques described herein. The media and programinstructions may be those specially designed and constructed for thepurposes of the present invention, or any kind well known and availableto those having skill in the computer software arts. When executed by aprocessor, these program instructions are suitable to implement any ofthe methods and techniques, and components thereof, described above.Examples of non-transitory computer-readable media include, but are notlimited to, magnetic media such as hard disks, semiconductor memory,optical media such as CD-ROM disks; magneto-optical media such asoptical disks; and hardware devices that are specially configured tostore program instructions, such as read-only memory devices (ROM),flash memory devices, EEPROMs, EPROMs, etc. and random access memory(RAM). Examples of program instructions include both machine code, suchas produced by a compiler, and files containing higher-level code thatmay be executed by the computer using an interpreter.

The foregoing description, for purposes of explanation, used specificnomenclature to provide a thorough understanding of the invention.However, it will be apparent to one skilled in the art that the specificdetails are not required in order to practice the invention. Thus, theforegoing descriptions of specific embodiments of the present inventionare presented for purposes of illustration and description. They are notintended to be exhaustive or to limit the invention to the precise formsdisclosed. It will be apparent to one of ordinary skill in the art thatmany modifications and variations are possible in view of the aboveteachings.

While the embodiments have been described in terms of several particularembodiments, there are alterations, permutations, and equivalents, whichfall within the scope of these general concepts. It should also be notedthat there are many alternative ways of implementing the methods andapparatuses of the present embodiments. It is therefore intended thatthe following appended claims be interpreted as including all suchalterations, permutations, and equivalents as fall within the truespirit and scope of the described embodiments.

What is claimed is:
 1. A mobile computing device for use in basketball,comprising a digital camera for capturing a plurality of two-dimensional(2D) images; and at least one processor programmed with instructionsthat, when executed by the at least one processor, cause the at leastone processor to: identify at least one of a basketball hoop and abackboard coupled to the basketball hoop within the plurality of 2Dimages based on a shape of the basketball hoop or the backboard withinthe plurality of 2D images; determine an orientation of the basketballhoop in three-dimensional (3D) space based on a shape of the basketballhoop or the backboard within the plurality of 2D images; determine alocation of the basketball hoop in 3D space based on the plurality of 2Dimages; identify a basketball within the plurality of 2D images; whenthe basketball is launched toward the basketball hoop, determine aplurality of 2D locations of the basketball along a trajectory of thebasketball toward the basketball hoop based on the plurality of 2Dimages; determine the trajectory of the basketball in 3D space based onthe plurality of 2D locations and at least one of a size of thebasketball and distance of the basketball from at least one objectwithin the plurality of 2D images; determine a parameter indicative ofshooting performance of a user in launching the basketball toward thebasketball hoop based on the determined trajectory, the determinedorientation of the basketball hoop in 3D space, and the determinedlocation of the basketball hoop in 3D space; and based on the parameter,provide feedback indicative of the shooting performance.
 2. The mobilecomputing device of claim 1, wherein the mobile communication device isa smart phone.
 3. The mobile computing device of claim 1, wherein themobile communication device is a tablet computer.
 4. The mobilecomputing device of claim 1, wherein the instructions, when executed bythe at least one processor, cause the at least one processor todetermine, based on the determined trajectory, an angle at which thebasketball is launched along the trajectory, and wherein the feedbackindicates the angle.
 5. The mobile computing device of claim 1, whereindetermination of the position of the at least one object in 3D space isbased on a size of the at least one object in the plurality of 2D imagesand a size for the at least one object specified by a user input.
 6. Themobile computing device of claim 1, wherein the instructions, whenexecuted by the at least one processor, cause the at least one processorto: identify the at least one object in the plurality of 2D images;determine a position of the at least one object in 3D space; determine aposition of the basketball in 3D space for at least one of the 2Dlocations based on the determined position of the at least one object in3D space and a location of the basketball relative to the at least oneobject in the plurality of 2D images; and determine the trajectory ofthe basketball in (3D) space based on the position of the basketball in3D space.
 7. The mobile computing device of claim 6, whereindetermination of the position of the at least one object in 3D space isbased on a size of the at least one object in the plurality of 2D imagesand a size for the at least one object specified by a user input.
 8. Themobile computing device of claim 6, wherein the at least one object is amarking on a basketball court for the basketball hoop.
 9. The mobilecomputing device of claim 6, wherein the at least one object is theuser.
 10. The mobile computing device of claim 1, wherein the parameteris an angle of entry of the basketball into the basketball hoop.
 11. Themobile computing device of claim 10, wherein the feedback indicates theangle.
 12. The mobile computing device of claim 1, wherein the parameteris a location of the basketball relative to the basketball hoop.
 13. Themobile computing device of claim 1, wherein the feedback indicates thelocation.
 14. The mobile computing device of claim 1, wherein theinstructions, when executed by the at least one processor, cause the atleast one processor to determine motions of a body part of the user. 15.The mobile computing device of claim 14, wherein the feedback isindicative of the determined motions.
 16. The mobile computing device ofclaim 14, wherein the instructions, when executed by the at least oneprocessor, cause the at least one processor to determine a statisticalassociation between basketball trajectory and body part motion of theuser.
 17. The mobile computing device of claim 16, wherein the feedbackis indicative of a consistency of at least one of the trajectory of thebasketball and the body part motion.
 18. A method for use in basketball,comprising: capturing a plurality of two-dimensional (2D) images with adigital camera; identifying at least one of a basketball hoop and abackboard coupled to the basketball hoop within the plurality of 2Dimages with at least one processor based on a shape of the basketballhoop or the backboard within the plurality of 2D images; determining,with the at least one processor, an orientation of the basketball hoopin three-dimensional (3D) space based on a shape of the basketball hoopor the backboard within the plurality of 2D images; determining, withthe at least one processor, a location of the basketball hoop in 3Dspace based on the plurality of 2D images; identifying, with the atleast one processor, a basketball within the plurality of 2D images;when the basketball is launched toward the basketball hoop, determiningwith the at least one processor a plurality of 2D locations of thebasketball along a trajectory of the basketball toward the basketballhoop based on the plurality of 2D images; determining, with the at leastone processor, the trajectory of the basketball in 3D space based on theplurality of 2D locations and at least one of a size of the basketballand a distance of the basketball from at least one object within theplurality of 2D images; determine a parameter indicative of shootingperformance of a user in launching the basketball toward the basketballhoop based on the determined trajectory, the determined orientation ofthe basketball hoop in 3D space, and the determined location of thebasketball hoop in 3D space; and based on the parameter, providingfeedback to the user, the feedback indicative of the shootingperformance.
 19. The method of claim 17, further comprising determining,with the at least one processor based on the determined trajectory, anangle at which the basketball is launched along the trajectory, whereinthe feedback indicates the angle.
 20. The method of claim 18, furthercomprising: identifying, with the at least one processor, the at leastone object in the plurality of 2D images; determining, with the at leastone processor, a position of the at least one object in 3D space;determining, with the at least one processor, a position of thebasketball in 3D space for at least one of the 2D locations based on thedetermined position of the at least one object in 3D space and alocation of the basketball relative to the at least one object in theplurality of 2D images; and determining, with the at least oneprocessor, the trajectory of the basketball in 3D space based on theposition of the basketball in 3D space.
 21. The method of claim 20,further comprising receiving, with the at least one processor, a userinput indicating a size of the at least one object, wherein thedetermining the position of the at least one object in 3D space is basedon a size of the at least one object in the plurality of 2D images andthe size of the at least one object indicated by the user input.
 22. Themethod of claim 18, wherein the at least one object is a marking on abasketball court for the basketball hoop.
 23. The method of claim 18,wherein the at least one object is the user.
 24. The method of claim 18,further comprising using the feedback in a video game.
 25. The method ofclaim 18, further comprising generating a training regimen based on thefeedback.